Four Types of Noise in Data for PAC Learning

In order to be useful in practice, machine learning algorithms must tolerate noisy inputs. In this paper we compare and contrast the effects of four different types of noise on learning in Valiant’s PAC (probably approximately correct), or distribution-free, model of learning [ 111. Two previously studied models, malicious noise [ 121 and random classification noise [ 11, represent the extremes. Malicious noise is intended to model the worst possible sort of noise, and in general only a very small amount of it can be tolerated [7]. On the other hand, Angluin and Laird [ l] have shown that for random misclassification noise instances are never altered but their labels are reversed with probability Y PAC learning can be achieved for any Y < l/2. They further show that any algorithm that chooses as its output concept some concept that minimizes disagreements with a polynomial size set of examples meets this bound. Here we extend Angluin and Laird’s result to malicious misclassification noise the noisy label is chosen adversarially instead of randomly. We also show that if one considers only algorithms that work by min-